cl-waffe

<APIs: cl-waffe>

!normal

!normal(dims &optional (mean 2.0) (stddev 1.0))
Initializes the new tensor with sampling the standard distribution.

REPL: 

CL-WAFFE> (!normal `(10 10)) :mean 2.0 :stddev 1.0

#Const(((0.939... 1.594... ~ 3.769... 2.525...)        
                 ...
        (2.022... 2.286... ~ 0.844... 3.609...)) :dtype :float :shape (10 10) :backward NIL)

!randn

!randn(dims)
Initializes the new tensor of dims with sampling normal distribution where mean=0.0, stddev=1.0

REPL: 

CL-WAFFE> (!randn `(10 10))

#Const(((-0.20... -0.17... ~ -0.82... 0.113...)        
                 ...
        (-1.33... -0.45... ~ 0.163... 2.218...)) :dtype :float :shape (10 10) :backward NIL)

!uniform-random

!uniform-random(dims &key (limit 1))

Initializes tensor with sampling uniform random.

The returned tensor is filled by random numbers 0<=x<limit

REPL: 

CL-WAFFE> (!uniform-random `(10 10) :limit 2.0)

#Const(((0.290... 1.492... ~ 1.814... 0.777...)        
                 ...
        (0.194... 1.930... ~ 1.669... 1.442...)) :dtype :float :shape (10 10) :backward NIL)

!beta

!beta(dims alpha beta)

Initializes the new tensor of dims with sampling beta distribution.

Algorithm: https://dl.acm.org/doi/pdf/10.1145/359460.359482

REPL: 

CL-WAFFE> (!beta `(10 10) 5.0 1.0)

#Const(((0.917... 0.964... ~ 0.927... 0.807...)        
                 ...
        (0.876... 0.777... ~ 0.902... 0.466...)) :dtype :float :shape (10 10) :backward NIL)

!gamma

!gamma(dims k &optional (theta 1.0))
Initializes the new tensor of dims with samples of gamma distribution.

REPL: 

CL-WAFFE> (!gamma `(10 10) 1.0)

#Const(((2.338... 1.112... ~ 0.845... 0.096...)        
                 ...
        (0.185... 0.081... ~ 0.041... 2.381...)) :dtype :float :shape (10 10) :backward NIL)

!chisquare

!chisquare(dims df)
Initializes tensor with samples of chi-square distribution using the gamma distribution..

REPL: 

CL-WAFFE> (!chisquare `(10 10) 2.0)

#Const(((0.228... 1.644... ~ 3.603... 1.272...)        
                 ...
        (8.122... 0.070... ~ 3.789... 0.322...)) :dtype :float :shape (10 10) :backward NIL)

!bernoulli

!bernoulli(dims rate)
Initializes the tensor of dims with sampling bernoulli distribution, where p=rate. p=[0, 1]

REPL: 

CL-WAFFE> (!bernoulli `(10 10) 0.5)

#Const(((1.0 1.0 ~ 1.0 0.0)        
                 ...
        (0.0 0.0 ~ 1.0 1.0)) :dtype :float :shape (10 10) :backward NIL)

!binomial

!binomial(dims rate)
Alias for !bernoulli

REPL: 

CL-WAFFE> (!binomial `(10 10) 0.5)

#Const(((0.0 0.0 ~ 0.0 0.0)        
                 ...
        (1.0 1.0 ~ 0.0 0.0)) :dtype :float :shape (10 10) :backward NIL)

!random-with

!random-with(dims f)

Initializes the tensor of dims. Each element is initialized with f, f is a funcallable function. and called with the index of the tensor.

See also: !init-with which is alias for !random-with.

REPL: 

CL-WAFFE> (!random-with '(10 10) #'(lambda (n) n))

#Const(((0.0 1.0 ~ 8.0 9.0)        
                 ...
        (90.0 91.0 ~ 98.0 99.0)) :dtype :float :shape (10 10) :backward NIL)

!random

!random(dims limit)
Initializes the new tensor of dims. Each element is consisted of a uniform-random within limit. limit must be following: fixnum, single-float, cons. and depending on this !random has a multiple behaviours.

REPL: 

CL-WAFFE> (!random `(10 10) 1.0)

#Const(((0.569... 0.154... ~ 0.840... 0.956...)        
                 ...
        (0.344... 0.495... ~ 0.212... 0.374...)) :dtype :float :shape (10 10) :backward NIL)

CL-WAFFE> (!random `(10 10) 3)

#Const(((2.0 1.0 ~ 0.0 2.0)        
                 ...
        (2.0 2.0 ~ 1.0 0.0)) :dtype :float :shape (10 10) :backward NIL)

CL-WAFFE> (!random `(10 10) `(1.0 2.0))

#Const(((1.642... 1.500... ~ 1.848... 1.415...)        
                 ...
        (1.032... 1.951... ~ 1.362... 1.905...)) :dtype :float :shape (10 10) :backward NIL)

CL-WAFFE> (!random `(10 10) `(1 5))

#Const(((2.0 2.0 ~ 4.0 4.0)        
                 ...
        (1.0 3.0 ~ 2.0 3.0)) :dtype :float :shape (10 10) :backward NIL)

!zeros-like

!zeros-like(tensor)

Return a const where the shape is the same as tensor but elements are zero.

Example: 

(setq a (!randn `(10 10)))
(!zeros-like a)
;#Const(((0.0 0.0 ~ 0.0 0.0)        
;                 ...
;        (0.0 0.0 ~ 0.0 0.0)) :mgl t :shape (10 10))

!ones-like

!ones-like(tensor)
Return a const where the shape is the same as tensor but elements are one. Example: 
(setq a (!randn `(10 10)))
(!ones-like a)
;#Const(((1.0 1.0 ~ 1.0 1.0)        
;                 ...
;        (1.0 1.0 ~ 1.0 1.0)) :mgl t :shape (10 10))

!full-like

!full-like(tensor element)
Return a const where the shape is the same as tensor but elements are specified value by element. Example: 
(setq a (!randn `(10 10)))
(!full-like a 3)
;#Const(((3.0 3.0 ~ 3.0 3.0)        
;                 ...
;        (3.0 3.0 ~ 3.0 3.0)) :mgl t :shape (10 10))

!zeros

!zeros(shape)

Initializing constant tensor with given shape, where initial elements are zero.

Input: shape (cons)

Output: Tensor (which is constant)

Example: 

(!zeros `(10 10))
;#Const(((0.0 0.0 ~ 0.0 0.0)        
;                ...
;        (0.0 0.0 ~ 0.0 0.0)) :mgl t :shape (10 10))

!ones

!ones(shape)

The same as !zeros but initial element is one.

Example: 

(!ones `(10 10))
;#Const(((1.0 1.0 ~ 1.0 1.0)        
;                ...
;        (1.0 1.0 ~ 1.0 1.0)) :mgl t :shape (10 10))

!fill

!fill(shape element)

The same as !zeros, !ones but initial element is given element.

Note: the argument element coerced into mgl-mat:*default-mat-ctype*

Example: 

(!fill '(10 10) 10)
;#Const(((10.0 10.0 ~ 10.0 10.0)        
;                  ...
;        (10.0 10.0 ~ 10.0 10.0)) :mgl t :shape (10 10))

!shape

!shape(tensor &optional (nth nil))

Returns the shape of tensor when nth=nil.nth indicates the index of shape, !shape return specified value.

Example: 

(setq a (!randn `(10 10 10)))
(!shape a) ; => (10 10 10)
(!shape a 0) ;=> 10

!dims

!dims(tensor)

Returns the total length of a given tensor's dims

Example: 

(!dims (!zeros '(10 10 10))) ; => 3

!size

!size(tensor)

Returns the total size of a tensor

Example: 

(!size (!zeros '(10 10 10))) ; => 1000

WaffeTensor

waffetensor
OptionValue
Constructor:(sysconst value &key (backend *default-backend*) (extend nil) (thread-data nil) (path-through-node? nil) (no-jit nil) (breakme? nil) &aux (data (init-waffe-tensor-data value)) (backend (check-backend backend extend)) (grad nil) (thread-data thread-data) (destructive? t) (is-next-destruct? breakme?) (is-sysconst? t) (force-ignore-jit no-jit) (path-through-node? path-through-node?) (is-mat (typep value (quote mat))) (grad-tmp (make-grad-tmp)))
Predicate:waffetensor-p
Copier:copy-waffetensor
Print Function:(lambda (tensor stream depth) (declare (ignore depth)) (format stream (render-tensor tensor)))

An structure of Waffe's Tensor. This structure have:

  1. data (type of WaffeTensorContentType)
  2. the computation node for backprops, and grads
  3. backend informations and parameters for optimizing.

There's three ways to make it.

(const value)
Constant tensor, grad won't be created.
(tensor value)
Parameter tensor, grad will be created.
(sysconst value)
Constant tensor where tensor sometime cached. Users don't have to use this.

Value is following:

  1. simple-array
  2. mgl-mat:mat (recommended)
  3. fixnum
  4. float
  5. null
  6. cons
  7. function (for lazy evaluation)
  8. ratio (when make, coerced to float)

This structure is printable and printed nicely.

  • data
    OptionValue
    Type:cl-waffe::waffetensortypes
    Read Only:nil
    Accessor:cl-waffe::waffetensor-data
    Initform:nil
  • grad-tmp
    OptionValue
    Type:cl-waffe::grad-tmp
    Read Only:nil
    Accessor:cl-waffe::waffetensor-grad-tmp
    Initform:(cl-waffe::make-grad-tmp)
  • backward
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe::waffetensor-backward
    Initform:nil
  • backend
    OptionValue
    Type:keyword
    Read Only:nil
    Accessor:cl-waffe::waffetensor-backend
    Initform::mgl
  • grad
    OptionValue
    Type:cl-waffe::waffetensortypes
    Read Only:nil
    Accessor:cl-waffe::waffetensor-grad
    Initform:nil
  • variables
    OptionValue
    Type:list
    Read Only:nil
    Accessor:cl-waffe::waffetensor-variables
    Initform:nil
  • state
    OptionValue
    Type:t
    Read Only:nil
    Accessor:cl-waffe::waffetensor-state
  • is-mat
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe::waffetensor-is-mat
    Initform:nil
  • is-param?
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe::waffetensor-is-param?
    Initform:nil
  • is-ancestor-param
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe::waffetensor-is-ancestor-param
    Initform:nil
  • is-next-destruct?
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe:waffetensor-is-next-destruct?
    Initform:nil
  • destructive?
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe:waffetensor-destructive?
    Initform:nil
  • thread-data
    OptionValue
    Type:(or cl-waffe::waffenodethread null)
    Read Only:nil
    Accessor:cl-waffe:waffetensor-thread-data
    Initform:nil
  • is-sysconst?
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe::waffetensor-is-sysconst?
    Initform:nil
  • path-through-node?
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe::waffetensor-path-through-node?
    Initform:nil
  • tensor-ident
    OptionValue
    Type:(or null symbol)
    Read Only:nil
    Accessor:cl-waffe::waffetensor-tensor-ident
    Initform:nil
  • force-ignore-jit
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe::waffetensor-force-ignore-jit
    Initform:nil
  • key
    OptionValue
    Type:(or null cons)
    Read Only:nil
    Accessor:cl-waffe::waffetensor-key
    Initform:nil
  • idx
    OptionValue
    Type:(or null symbol)
    Read Only:nil
    Accessor:cl-waffe::waffetensor-idx
    Initform:nil
  • is-data-destructed?
    OptionValue
    Type:boolean
    Read Only:nil
    Accessor:cl-waffe:waffetensor-is-data-destructed?
    Initform:nil

parameter

parameter(tensor)

Redefining new-tensor where old-tensor is const or tensor.

The new-tensor can made grads.

Excepted usage is like: 

(setq my-param (parameter (!mul 0.01 (!randn `(10 10)))))

Note that: tensor's computation node that old-tensor has, will be lost. Only tensor's data and backend will be extended.

Input
Tensor (as usual, defined by (const)(sysconst)(tensor))
Output
Tensor (as usual, defined by (tensor))

data

data(tensor)

Access tensor's data. This won't be copied.

When tensor's data is lazy evaluted, this function behave following:

  1. When tensor is transposed and lazy evaluted, directly returns function object for speed.
  2. When tensor is cached and lazy evaluted, returns mat object.
Input
WaffeTensor
Output
mgl-mat:mat, or waffetensorcontentdata

when (data tensor) is a function and is:

cached mat
Return mgl-mat, this do not make copy
lazy-evaluation or transposed
Return function itself

Note: this function is setfable and inlined

value

value(tensor &key (ignore-transpose nil))

Access tensor's data, but if tensor is lazy-evaluated, eval them.

Note: this is not setfable

backward

backward(tensor)

Compute back propagation by traversing the Tensor's computation node.

The parameters of the model defined by (tensor) or to which (Parameter tensor) is applied, store the gradient in grad slot.

Note that: tensor must be the shape of `(1) or single value. Otherwise an error occurs.

In the process calculating backward, new backwards won't be created. (*no-grad* automatically becomes t)

Input
WaffeTensor
Output
NIL

with-no-grad

with-no-grad(&body body)
Below this macro, the parameter *no-grad* become t, which means: some operations are forcibly ignored. (e.g.: save-for-backward, building computation nodes) 
(with-no-grad
  (call (model) x))

*no-grad*

*no-grad*
When t, some node will be ignored. see references below for details. default: nil

!add

!add(x y)

Adds x and y.

In the case when x or y is not a tensor, automatically creates a new tensor.

Destructive mode: (!!add x y)

It supports:

  1. Broadcasting shapes
  2. JIT

Examples

(setq a (!randn `(3 3)))
(setq b (!randn `(3 3)))
(setq c (!randn `(3 1)))

(!add 1 1)
;=> Const(2)

(!add (const 1)(const 1))
;=> Const(2)

(!add a b)
;#Const(((3.418... 1.974... 0.177...)
;                 ...
;        (-1.30... 0.987... 1.917...)) :mgl t :shape (3 3))

(!add a c)
;#Const(((1.426... 2.129... 1.050...)
;                 ...
;        (-0.64... 0.269... 0.303...)) :mgl t :shape (3 3))

!!add

!!add(target-x y)

Adds target-x and y in a destructive way.

target-x is always substituted for the result

y is not subject to side effects unless target-x is not a mat.

See also: Destructive Operations

!sub

!sub(x y)

Subtract x by y.

In the case when x or y is not a tensor, automatically creates a new tensor.

It supports:

  1. Broadcasting shapes
  2. JIT

Examples

(setq a (!randn `(3 3)))
(setq b (!randn `(3 3)))
(setq c (!randn `(3 1)))

(!sub 1 1)
;=> Const(0)

(!sub (const 1)(const 1))
;=> Const(0)

(!sub a b)
;#Const(((-0.86... 1.413... 1.139...)
;                 ...
;        (0.017... -0.44... -1.31...)) :mgl t :shape (3 3))

(!sub a c)
;#Const(((1.128... 1.258... 0.267...)
;                 ...
;        (-0.64... 0.269... 0.303...)) :mgl t :shape (3 3))

!!sub

!!sub(target-x y)

Substracts target-x by y in a destructive way.

target-x is always substituted for the result.

y is not subject to side effects unless target-x is not a mat.

See also: Destructive Operations

!mul

!mul(x y)

Multiply x and y with element-wise.

In the case when x or y is not a tensor, automatically creates a new tensor.

It supports:

  1. Broadcasting shapes
  2. JIT

Examples

(setq a (!randn `(3 3)))
(setq b (!randn `(3 3)))
(setq c (!randn `(3 1)))

(!mul 1 1)
;=> Const(1)

(!mul (const 1)(const 1))
;=> Const(1)

(!mul a b)
;#Const(((2.734... 0.475... -0.31...)        
;                 ...
;        (0.426... 0.193... 0.490...)) :mgl t :shape (3 3))

(!mul a c)
;#Const(((2.734... 0.475... -0.31...)        
;                 ...
;        (0.426... 0.193... 0.490...)) :mgl t :shape (3 3))

!!mul

!!mul(target-x y)

Multiplys target-x and y in a destructive way.

target-x is always substituted for the result

y is not subject to side effects unless target-x is not a mat.

See also: Destructive Operations

!div

!div(x y)

Divides x by y.

In the case when x or y is not a tensor, automatically creates a new tensor.

It supports:

  1. Broadcasting shapes
  2. JIT

Examples

(setq a (!randn `(3 3)))
(setq b (!ones `(3 3)))
(setq c (!ones `(3 1)))

(!div 2 1)
;=> Const(2)

(!div (const 2)(const 1))
;=> Const(2)

(!div a b)
;#Const(((1.734... 0.475... -0.31...)        
;                 ...
;        (0.426... 0.193... 0.490...)) :mgl t :shape (3 3))

(!div a c)
;#Const(((2.734... 0.475... -0.31...)        
;                 ...
;        (0.426... 0.193... 0.490...)) :mgl t :shape (3 3))

!sum

!sum(x &optional (axis nil) (keepdims nil))

Sum up x where x is a cl-waffe tensor.

For nd tensors...

1D
unsqueeze x with 1, and call !sum again.
2D and more.
Sum up all elements of X

arguments

axis
a dimension to reduce
keepdims
When t, the returning tensor is repeated with axis

Example

(setq a (!randn `(10)))
(!sum a)
;=>#Const(4.74653)

(setq a (!randn `(10 10)))
(!sum a)
;=>#Const(1.5428619)

(!sum a 0)
;=>#Const(((-2.07... 0.463... ~ 1.778... 1.695...)) :mgl t :shape (1 10))

(!sum a 1)
;#Const(((0.967...)        
;                 ...
;        (2.774...)) :mgl t :shape (10 1))

(!sum a 0 t)
;#Const(((-2.07... 0.463... ~ 1.778... 1.695...)        
;                 ...
;        (-2.07... 0.463... ~ 1.778... 1.695...)) :mgl t :shape (10 10))

!mean

!mean(x &optional (axis nil) (keepdims nil))
The usage is the same as !sum.

Example

(setq a (!ones '(10 10)))
;#Const(((1.0 1.0 ~ 1.0 1.0)        
;                 ...
;        (1.0 1.0 ~ 1.0 1.0)) :mgl t :shape (10 10))
(!mean a)
;=>Const(1.0)

!dot

!dot(x y)

Computes the dot product of x and y where x and y are 1d Tensor.

🗒Note: Unlike Numpy's dot, !dot only supports for 1d tensors with the same number of elements and the tensor of which dims is larger than 1, regarded as 1d tensors.

Example

(setq a (!randn `(10)))
(setq b (!randn `(10)))

(!dot a b)
;=> #Const(1.0842022e-19)

!matmul

!matmul(x y)

Multiplying matrices x and y.

!matmul has many behaviours depends on the dimensionality of the tensors as follows:

x and y are 1D
The dot-product is returned. 
(setq a (!randn `(10)))
(setq b (!randn `(10)))
(!matmul a b)
;=>#Const(-2.0)
x and y are both 2D
The matrix-matrix product is returned. 
(setq a (!randn `(3 10)))
(setq b (!randn `(10 3)))
(!matmul a b)
;#Const(((2.309... 2.223... 3.630...)        
;                 ...
;        (2.334... 2.850... 3.678...)) :mgl t :shape (3 3))
x is 2D and y is 3D.
The matrix and y's each matrix are multiplied and is returned. 
(setq a (!randn `(3 10)))
(setq b (!randn `(5 10 3)))

(!matmul a b)
;(!aref b 0) ~ (!aref b 4) is multiplied with a

;#Const((((3.257... 2.731... 1.670...)         
;                   ...
;         (2.523... 2.251... 1.276...))        
;                 ...
;        ((2.610... 2.764... 2.415...)         
;                   ...
;         (2.080... 2.204... 1.751...))) :mgl t :shape (5 3 3))
x is 3D and y is 2D.
The matrix and x's each matrix are multiplied and is returned. 
(setq a (!randn `(5 3 10)))
(setq b (!randn `(10 3)))

(!matmul a b)
;(!aref a 0) ~ (!aref a 4) is multiplied with b
;#Const((((2.309... 2.204... 1.556...)         
;                   ...
;         (3.746... 3.869... 3.091...))        
;                 ...
;        ((3.260... 3.200... 2.847...)         
;                   ...
;         (3.008... 2.186... 2.376...))) :mgl t :shape (5 3 3))
x is 3D and y is 3D.
The Batch Filtered Matrix-Matrix product is returned. 
(setq a (!randn `(5 3 10)))
(setq b (!randn `(5 10 3)))

; The returned mat is comprised of:
; (!matmul (!aref a 0)(!aref b 0))
; (!matmul (!aref a 1)(!aref b 1))
; (!matmul (!aref a 2)(!aref b 2))
; (!matmul (!aref a 3)(!aref b 3))

(!matmul a b)
;#Const((((6.621... -5.61... 2.898...)         
;                   ...
;         (-2.96... -4.26... -3.99...))        
;                 ...
;        ((-0.02... 2.707... 5.989...)         
;                   ...
;         (-3.35... 3.561... -3.90...))) :mgl t :shape (5 3 3))
Otherwise
Currently not implemented. In the near future for more will be added.

!sin

!sin(x)
Applying sin to each element of x, creating a new sysconst.

Example

(setq a (!randn `(5)))
;=>#Const((0.638... 0.527... 0.515... 0.495... 0.912...) :mgl t :shape (5))
(!sin a)
;=>#Const((-0.44... -0.64... -0.66... -0.70... -0.09...) :mgl t :shape (5))

!cos

!cos(x)
Applying cos to each element of x, creating a new sysconst.

Example

(setq a (!randn `(5)))
;=>#Const((0.638... 0.527... 0.515... 0.495... 0.912...) :mgl t :shape (5))
(!cos a)
;=>#Const((0.803... 0.864... 0.870... 0.879... 0.611...) :mgl t :shape (5))

!tan

!tan(x)
Applying tan to each element of x, creating a new sysconst.

Example

(setq a (!randn `(5)))
;=>#Const((0.638... 0.527... 0.515... 0.495... 0.912...) :mgl t :shape (5))
(!tan a)
;=>#Const((0.741... 0.582... 0.566... 0.540... 1.293...) :mgl t :shape (5))

!asin

!asin(x)
Applying asin to each element

!acos

!acos(x)
Applying acos to each element

!atan

!atan(x)
Applying atan to each element

!sinh

!sinh(x)
Applying sinh to each element of x, creating a new sysconst.

Example

(setq a (!randn `(5)))
;=>#Const((0.638... 0.527... 0.515... 0.495... 0.912...) :mgl t :shape (5))
(!sinh a)
;=>#Const((0.682... 0.551... 0.538... 0.516... 1.044...) :mgl t :shape (5))

!cosh

!cosh(x)
Applying cosh to each element of x, creating a new sysconst.

Example

(setq a (!randn `(5)))
;=>#Const((0.638... 0.527... 0.515... 0.495... 0.912...) :mgl t :shape (5))
(!cosh a)
;=>#Const((1.210... 1.142... 1.135... 1.125... 1.446...) :mgl t :shape (5))

!tanh

!tanh(x)
Applying tanh to x, return a new sysconst with making nodes.

!asinh

!asinh(x)
Applying asinh to each element

!acosh

!acosh(x)
Applying acosh to each element

!atanh

!atanh(x)
Applying atanh to each element

!abs

!abs(x)

Computes the absolute value of each element in x.

Example: 

(setq a (!random `(10 10) '(-1.0 1.0)))
;#Const(((0.048... 0.805... ~ 0.769... 0.252...)        
;                 ...
;        (0.159... -0.66... ~ -0.55... -0.23...)) :mgl t :shape (10 10))
(!abs a)
;#Const(((0.048... 0.805... ~ 0.769... 0.252...)        
;                 ...
;        (0.159... 0.667... ~ 0.553... 0.239...)) :mgl t :shape (10 10))

!log

!log(x)

Returns a new tensor with the natural logarithm of the elements of input.

yi = log(e xi)

Example

(setq a (!ones '(10 10)))
(!log a)
;#Const(((0.0 0.0 ~ 0.0 0.0)        
;                 ...
;        (0.0 0.0 ~ 0.0 0.0)) :mgl t :shape (10 10))

!exp

!exp(x)
Applying exp to each element of x, creating a new sysconst.

Example

(setq a (!randn `(10 10)))
;#Const(((0.624... 0.807... ~ 0.500... 0.937...)        
;                 ...
;        (0.662... 0.299... ~ 0.761... 0.729...)) :mgl t :shape (10 10))
(!exp a)
;#Const(((1.866... 2.242... ~ 1.650... 2.553...)        
;                 ...
;        (1.939... 1.349... ~ 2.140... 2.073...)) :mgl t :shape (10 10))

!pow

!pow(x n)
Takes the power of each element in x with n, returning a new sysconst.

Example

(setq a (!ones `(10 10)))
(!pow a 3)
;#Const(((1.0 1.0 ~ 1.0 1.0)        
;                 ...
;        (1.0 1.0 ~ 1.0 1.0)) :mgl t :shape (10 10))

!sqrt

!sqrt(x)
Takes the power of each element in x with 1/2, creating new sysconst and nodes.

Example

(setq a (!ones `(10 10)))
(!sqrt a 3)
;#Const(((1.0 1.0 ~ 1.0 1.0)
;                 ...
;        (1.0 1.0 ~ 1.0 1.0)) :mgl t :shape (10 10))

!argmax

!argmax(tensor &key (dim -1) (keepdims nil) (max nil))

Returns the indices of the maximum value of all elements in the input tensor.

If max=t, retures the maximun value of dim.

dim
The dimension to reduce. If nil, the argmax of the flattened input is returned.
keepdims
whether the output tensor has dim retained or not. Ignored if dim=-1

Example

(setq a (!randn `(5)))
;#Const((0.933... 0.158... 0.822... 0.881... 0.831...) :mgl t :shape (5))
(!argmax a)
;#Const((0.0) :mgl t :shape (1))
(setq a (!randn `(10 10 10)))
;#Const((((0.393... 0.658... ~ 0.003... 0.609...)         
;                   ...
;         (0.394... 0.252... ~ 0.688... 0.057...))        
;                 ...
;        ((0.325... 0.794... ~ 0.540... 0.381...)         
;                   ...
;         (0.310... 0.035... ~ 0.280... 0.431...))) :mgl t :shape (10 10 10))

(!argmax a :dim 2)

;#Const(((5.0 9.0 ~ 0.0 4.0)        
;                 ...
;        (2.0 0.0 ~ 2.0 5.0)) :mgl t :shape (10 10))

(!argmax a :dim 2 :keepdims t)
;#Const((((5.0 5.0 ~ 5.0 5.0)         
;                   ...
;         (4.0 4.0 ~ 4.0 4.0))        
;                 ...
;        ((2.0 2.0 ~ 2.0 2.0)         
;                   ...
;         (5.0 5.0 ~ 5.0 5.0))) :mgl t :shape (10 10 10))

!argmin

!argmin(tensor &key (dim -1) (keepdims nil) (min nil))

Returns the indices of the minimum value of all elements in the input tensor.

If min=t, argmin returns the minimum value of dim.

dim
The dimension to reduce. If nil, the argmax of the flattened input is returned.
keepdims
whether the output tensor has dim retained or not. Ignored if dim=-1.

Example

(setq a (!randn `(5)))
;=>#Const((0.635... 0.101... 0.864... 0.563... 0.481...) :mgl t :shape (5))
(!argmin a)
;=>#Const((1.0) :mgl t :shape (1))

(setq a (!randn `(10 10 10)))
;#Const((((0.267... 0.113... ~ 0.142... 0.208...)         
;                   ...
;         (0.174... 0.948... ~ 0.232... 0.462...))        
;                 ...
;        ((0.454... 0.361... ~ 0.605... 0.731...)         
;                   ...
;         (0.099... 0.816... ~ 0.729... 0.996...))) :mgl t :shape (10 10 10))

(!argmin a)
;#Const((415.0...) :mgl t :shape (1))

!squeeze

!squeeze(x &optional (dim nil))

Returns a new tensor with a dimension of size one removed at the specified position.

When dim=nil or -1, the last position of dim will be removed.

If the specified position of a tensor isn't one, !squeeze is skipped.

Example

(setq a (!randn `(10 1 10)))
;#Const((((0.928... 0.556... ~ 0.697... 0.973...))        
;                 ...
;        ((0.368... 0.995... ~ 0.589... 0.716...))) :mgl t :shape (10 1 10))

(!squeeze a 1)
;#Const(((0.928... 0.556... ~ 0.697... 0.973...)        
;                 ...
;        (0.368... 0.995... ~ 0.589... 0.716...)) :mgl t :shape (10 10))

(!squeeze a -1)
;#Const((((0.928... 0.556... ~ 0.697... 0.973...))        
;                 ...
;        ((0.368... 0.995... ~ 0.589... 0.716...))) :mgl t :shape (10 1 10))

(setq a (!randn `(10 10 1)))
;#Const(((0.991... 0.248... ~ 0.610... 0.289...)        
;                 ...
;        (0.593... 0.177... ~ 0.374... 0.668...)) :mgl t :shape (10 10))

!unsqueeze

!unsqueeze(x &optional (dim 0) (count 1))

Returns a new tensor with a dimension of size one inserted at the specified position.

dim indicates the position, when dim=-1, it indicates a last dimension of x.

Example

(setq a (!randn `(10 10)))
;#Const(((0.685... 0.827... ~ 0.076... 0.102...)        
;                 ...
;        (0.802... 0.571... ~ 0.207... 0.283...)) :mgl t :shape (10 10))
(!unsqueeze a)
;#Const((((0.685... 0.827... ~ 0.076... 0.102...)         
;                   ...
;         (0.802... 0.571... ~ 0.207... 0.283...))) :mgl t :shape (1 10 10))

(!unsqueeze a -1)
;#Const((((0.685...)         
;                   ...
;         (0.102...))        
;                 ...
;        ((0.802...)         
;                   ...
;         (0.283...))) :mgl t :shape (10 10 1))

(!unsqueeze a 2)
;#Const(((0.685... 0.827... ~ 0.076... 0.102...)        
;                 ...
;        (0.802... 0.571... ~ 0.207... 0.283...)) :mgl t :shape (10 10 1 1))

!reshape

!reshape(x dim)

Return a new sysconst with changing its shape. x won't be modified.

If dims has the element of t, t is automatically inferred from the remaining dimensions and the number of elements in dim. (count t dim) must be 1 (Todo: Fix).

The total size of tensor must not be changed before or after the call to reshape.

See also: nil

Example

(setq a (!randn `(10 10 10)))
(!reshape a '(1 10 100))
;#Const((((0.454... 0.277... ~ 0.536... 0.135...)         
;                   ...
;         (0.857... 0.714... ~ 0.169... 0.279...))) :mgl t :shape (1 10 100))

(!reshape a '(1 1 t))
;#Const((((0.454... 0.277... ~ 0.169... 0.279...))) :mgl t :shape (1 1 1000))

!repeats

!repeats(x axis repeats)

Repeats x along specified axis by repeats, creating new sysconst.

x can be: mat or tensor.

Example

(setq a (!randn '(1 3 3)))
;#Const((((0.333... 0.914... 0.260...)         
;                   ...
;         (0.611... 0.110... 0.113...))) :mgl t :shape (1 3 3))
(!repeats a 0 3)
;#Const((((0.333... 0.914... 0.260...)         
;                   ...
;         (0.611... 0.110... 0.113...))
;                 ...
;        ((0.333... 0.914... 0.260...)         
;                   ...
;         (0.611... 0.110... 0.113...))) :mgl t :shape (3 3 3))

(!repeats (const 10.0) 3 10)
;#Const(((((10.0 10.0 ~ 10.0 10.0)))) :mgl t :shape (1 1 1 10))

!flatten

!flatten(tensor)

Flattens input by reshaping it into a one-dimensional tensor.

The operation is the same as (!reshape tensor '(t))

Example: 

(setq a (!randn `(10 10)))
;#Const(((0.688... 0.580... ~ 0.013... 0.461...)        
;                 ...
;        (0.214... 0.248... ~ 0.540... 0.416...)) :mgl t :shape (10 10))

(!flatten a)
;#Const((0.688... 0.580... ~ 0.540... 0.416...) :mgl t :shape (100))

!transpose

!transpose(x &optional result)

Transpose x where x is a 2d tensor.

Transposed x is lazy evaluated until called by !matmul.

Todo: implement 3d, 4d version...

Example

(setq a (!randn `(3 5)))
(setq a (!transpose a))
;#Const(#<FUNCTION (LABELS CL-WAFFE.BACKENDS.MGL::LAZYTRANSPOSE :IN CL-WAFFE.BACKENDS.MGL::LAZY-EVAL-TRANSPOSE) {10038CBADB}>)

(!matmul a (!randn '(3 5)))
;#Const(((0.653... 0.400... 0.471... 0.705... 0.623...)        
;                 ...
;        (1.220... 0.760... 0.975... 1.360... 1.029...)) :mgl t :shape (5 5))

!transpose1

!transpose1(x &rest result)

Transpose x but doesn't produce lazy-eval.

Todo: Numcl's operation couldm't optimized well. i need to reimplement it by myself.

Example

(setq a (!randn `(10 5 3)))

(!transpose1 a)
;#Const((((-0.47... -0.03... ~ -0.17... 0.328...)         
;                   ...
;         (0.210... -1.80... ~ 1.648... 0.135...))        
;                 ...
;        ((-0.52... 1.509... ~ 0.643... 0.258...)         
;                   ...
;         (-0.26... -1.14... ~ -1.08... 1.126...))) :mgl t :shape (3 5 10))

!concatenate

!concatenate(axis &rest tensors)
Concatenates the given sequence of tensors in the given axis. All tensors must have the same shape.

Example

(setq a (!randn `(3 3 3)))
;#Const((((1.000... -0.00... -0.25...)         
;                   ...
;         (1.473... -0.44... 1.680...))        
;                 ...
;        ((0.569... 0.852... 0.405...)         
;                   ...
;         (0.024... 0.756... 0.383...))) :mgl t :shape (3 3 3))

(!concatenate 0 a a a)
;#Const((((1.000... -0.00... -0.25...)         
;                   ...
;         (1.473... -0.44... 1.680...))        
;                 ...
;        ((0.569... 0.852... 0.405...)         
;                   ...
;         (0.024... 0.756... 0.383...))) :mgl t :shape (9 3 3))

(mgl-mat:M= (data (!aref * '(0 3)))
            (data (!aref * '(3 6))))
;T

!stack

!stack(axis &rest tensors)

Stacks the given tensors in the specified axis.

Internally, !stack adds 1 to the specified axis before calling !concatenate.

Note: Currently, when unsqueezing given tensors, !stack creates copies every time in order to prevent side effects. To avoid this, !concatenate is recommended to use. (TO FIX)

Example

(setq a (!randn `(2 2 2)))

;#Const((((-0.83... -1.74...)
;         (0.119... 0.162...))
;        ((-1.81... 0.907...)
;         (-0.50... -0.96...))) :mgl t :shape (2 2 2))

(!stack 0 a a a)
;#Const(((((-0.83... -1.74...)
;          (0.119... 0.162...))
;         ((-1.81... 0.907...)
;          (-0.50... -0.96...)))        
;                 ...
;        (((-0.83... -1.74...)
;          (0.119... 0.162...))
;         ((-1.81... 0.907...)
;          (-0.50... -0.96...)))) :mgl t :shape (3 2 2 2))

(mgl-mat:M= (data (!aref * 0))(data (!aref * 1)))
; T

!split

!split(tensor split-size &key (axis 0))

Splits the tensor into chunks in the specified axis. Each chunk is a copy of original tensor.

split-size indicates the strides of each chunk, that is, tensor will be split into equalliy size of split-size.

split-size must be fixnum.rr

Alternatively, !aref, (setf !aref) is available.

Example

(setq a (!randn `(4 2 2)))
;#Const((((-0.48... -1.22...)
;         (0.251... 0.476...))        
;                 ...
;        ((-0.66... 1.045...)
;         (-0.44... 1.592...))) :mgl t :shape (4 2 2))

(!split a 2)
;(#Const((((-0.48... -1.22...)
;         (0.251... 0.476...))
;        ((0.864... -0.93...)
;         (-0.43... 0.346...))) :mgl t :shape (2 2 2))
; #Const((((-1.91... -0.63...)
;         (-0.08... 0.867...))
;        ((-0.66... 1.045...)
;         (-0.44... 1.592...))) :mgl t :shape (2 2 2)))

; the rests are filled with 0.0
(!split a 3)
;(#Const((((-0.48... -1.22...)
;         (0.251... 0.476...))        
;                 ...
;        ((-1.91... -0.63...)
;         (-0.08... 0.867...))) :mgl t :shape (3 2 2))
; #Const((((-0.66... 1.045...)
;         (-0.44... 1.592...))        
;                 ...
;        ((0.0 0.0)
;         (0.0 0.0))) :mgl t :shape (3 2 2)))

!hstack

!hstack(&rest tensors)
!vstack is the equivalent to !concatenate(axis=1)

!vstack

!vstack(&rest tensors)
!vstack is the equivalent to !concatenate(axis=0)

!aref

!aref(tensor &rest dims)

!aref creates a new tensor from the area specified by dims from the given tensor.

This function is setfable and both function produces the computation nodes.

dims is consisted of list, and each dimension is described as follow formats:

t
t means (0~max-len) in the dimension.
fixnum
copies the index of fixnum in the dimension.
list
list must be of (start stop), copying tensors from start to stop in the dimension. that is, the result in the dimension is the copy of: start<=x<stop. Using t as stop means: t is the last element in the dimension.

The fixnum used in dims is not only positive numbers but also negative numbers.

For example, -1 is interpreted as (+ maxlen -1), -2 is interpreted as (+ maxlen -2)...

Note: (setf !aref) overwrites the given tensor's mat but won't overwrites its computation node. in order to update nodes, you must write it like: (setq a (setf (!aref a ...) ...))... See Example for the details.

Tensor cut-outs act on:

When is not setf
act on the given tensor.
When is setf
act on the target tensor. (e.g.: (setf (!aref target-tensor ...) input-tensor))

Example: 

(setq a (!randn `(10 5 3)))
;#Const((((0.621... -1.15... 2.396...)         
;                   ...
;         (0.157... 0.389... 1.084...))        
;                 ...
;        ((1.123... -0.58... -0.28...)         
;                   ...
;         (0.506... -0.44... -0.26...))) :mgl t :shape (10 5 3))

(!aref a '(0 3)) ; interpreted as (!aref a '(0 3) t t)
;#Const((((0.621... -1.15... 2.396...)         
;                   ...
;         (0.157... 0.389... 1.084...))        
;                 ...
;        ((0.694... 0.954... 1.210...)        
;                   ...
;         (0.884... 0.059... 0.190...))) :mgl t :shape (3 5 3))

(!aref a '(1 3))
;#Const((((0.657... 0.834... -2.01...)         
;                   ...
;         (1.194... 0.517... 0.356...))
;        ((0.694... 0.954... 1.210...)         
;                   ...
;         (0.884... 0.059... 0.190...))) :mgl t :shape (2 5 3))

(!aref a '(1 0)) ; When (cdr dims) <= 0, interpreted as (- (!shape tensor dim)(cdr dims))
; In this Example, this is the same as (!aref a '(1 10))
;#Const((((0.657... 0.834... -2.01...)         
;                   ...
;         (1.194... 0.517... 0.356...))        
;                 ...
;        ((1.123... -0.58... -0.28...)         
;                   ...
;         (0.506... -0.44... -0.26...))) :mgl t :shape (9 5 3))

(!aref a '(1 -1))
;#Const((((0.657... 0.834... -2.01...)         
;                   ...
;         (1.194... 0.517... 0.356...))        
;                 ...
;        ((-2.29... -1.12... -0.68...)         
;                   ...
;         (-1.74... 0.489... 1.519...))) :mgl t :shape (8 5 3))

(!aref a t '(0 2))
;Tensors in lower dimensions can also be clipped.
;If 0th dim isn't needed to be cut, place t.
;#Const((((0.621... -1.15... 2.396...)
;         (0.642... 0.029... 1.334...))        
;                 ...
;        ((1.123... -0.58... -0.28...)
;         (-2.43... -0.29... 0.882...))) :mgl t :shape (10 2 3))

(!aref a '(0 2) '(1 2) '(1 3))
;#Const((((0.029... 1.334...))
;        ((-1.41... -0.32...))) :mgl t :shape (2 1 2))

; This function is setfable, but currently I won't come up with the best solution to update computation node.
; I know it is very ugly but additional setq is required after setf.
; Also, note that (setf !aref). overwrites a.
(setq a (setf (!aref a '(0 3) '(0 3))(!zeros '(3 3))))

;#Const((((0.0 0.0 0.0)         
;                   ...
;         (0.157... 0.389... 1.084...))
;                 ...
;        ((1.123... -0.58... -0.28...)
;                   ...
;         (0.506... -0.44... -0.26...))) :mgl t :shape (10 5 3))

(!aref a 0 0)
;#Const((((0.0 0.0 0.0))) :mgl t :shape (1 1 3))

!where

!where(condition tensor then else)

Return a tensor of elements selected from either x or y, depending on condition.condition is given as a lambda expression, which called with an value of (aref tensor index).

!where defined asout = if (condition(tensor[i]), then, else)

Return: A tensor of shape that equal to the condition.

Example

(setq a (!random `(10 10) '(-1.0 1.0)))
;#Const(((0.042... -0.36... ~ 0.250... 0.967...)        
;                 ...
;        (-0.21... 0.962... ~ -0.32... 0.215...)) :mgl t :shape (10 10))

(!where #'(lambda (x)(> x 0)) a 1.0 0.0)
;#Const(((1.0 0.0 ~ 1.0 1.0)        
;                 ...
;        (0.0 1.0 ~ 0.0 1.0)) :mgl t :shape (10 10))

; works as ReLU

(!mul a (!where #'(lambda (x)(> x 0)) a 1.0 0.0))
;#Const(((0.042... 0.0... ~ 0.250... 0.967...)        
;                 ...
;        (0.0... 0.962... ~ 0.0... 0.215...)) :mgl t :shape (10 10))

!index

!indexnil
Todo

!filter

!filter(tensor lambda)
Applying every tensor's element lambda, it returns an tensor which comprised of the lambda's returned values.
tensor
an tensor that to be refered to
lambda
an function that returns elements at position x
(setq tensor (!randn `(10 10)))
(!filter tensor #'(lambda (x)(if (> x 0) x 1.0)))
;#Const(((0.802... 1.331... ~ 0.998... 1.994...)        
;                 ...
;        (1.0 0.005... ~ 0.296... 0.358...)) :mgl t :shape (10 10))

!arange

!arange(&rest args)

Like numpy's arange, arange can be called with a varying number of positional arguments:

(!arange stop)

(!arange 10)
;#Const((0.0 1.0 ~ 8.0 9.0) :mgl t :shape (10))

(!arange start stop)

(!arange 3 10)
;=>#Const((3.0 4.0 ~ 8.0 9.0) :mgl t :shape (7))

(!arange start stop step)

(!arange 3 10 2)
;#Const((3.0 5.0 7.0 9.0) :mgl t :shape (4))

!relu

!relu(x)

Applying relu to x, return a new sysconst with making nodes.

Relu(x) = { 0 (x < 0), x (x > 0) }

Input: x where x is waffe supported data type.

Output: Tensor

!sigmoid

!sigmoid(x)

Applyong sigmoid to x, return a new sysconst with making nodes.

Input: x where x is waffe supported data type.

Output: Tensor

!gelu

!gelu(x &key (approximate t))

Applying gelu to x, returning a new sysconst.

Paper: https://arxiv.org/abs/1606.08415.

TOOD: Improve its performance

GeLU(x) = x * s(x)

When approximate is t:

s(x) = x/2 * [1 + tanh(sqrt(2/pi * (x + 0.044715 * x^3)))]

When is nil:

Not implemented (TODO)

(setq x (!randn `(10 10)))
(!gelu x)
;#Const(((0.201... 0.038... ~ 0.158... 0.040...)        
;                 ...
;        (0.300... 1.395... ~ 0.030... 0.029...)) :mgl t :shape (10 10))

!leakey-relu

!leakey-relu(x &optional (alpha 0.01))

Applying Leakey-relu to x, returning a new sysconst.

Leakey-ReLU is defined as out = {alpha (x < 0), x (x >= 0)}

Example:

(setq x (!randn `(10 10)))
#Const(((0.635... -0.56... ~ -1.15... -1.50...)        
                 ...
        (0.775... 1.258... ~ -1.29... 0.240...)) :mgl t :shape (10 10))

(!leakey-relu x)
#Const(((0.635... 0.003... ~ 0.013... 0.022...)        
                 ...
        (0.775... 1.258... ~ 0.016... 0.240...)) :mgl t :shape (10 10))

!swish

!swish(x &key (beta (const 1.0)))

Applying swish to each element of x

Swish is defined as out = (/ 1 (+ 1 (exp (* beta -1 x))))

In default beta is 1.0, if you want to use trainable one, Swish is available as a waffe model.

Note that beta must begin given as a waffetensor.

(setq x (!randn `(10 10)))
#Const(((0.635... -0.56... ~ -1.15... -1.50...)        
                 ...
        (0.775... 1.258... ~ -1.29... 0.240...)) :mgl t :shape (10 10))

(!swish x)
;#Const(((0.415... -0.20... ~ -0.27... -0.27...)        
;                 ...
;        (0.531... 0.980... ~ -0.27... 0.134...)) :mgl t :shape (10 10))

(call (Swish :beta 1.0) x) ; its beta is trainable by backpropgating.
;#Const(((0.415... -0.20... ~ -0.27... -0.27...)        
;                 ...
;        (0.531... 0.980... ~ -0.27... 0.134...)) :mgl t :shape (10 10))

!softmax

!softmax(x &key (avoid-overflow t))

Applying softmax to x. !softmax has three behaviours depending on the number of dimensions.

The number of dims is...

1
Softmax is applied to dim=0 
(setq a (!randn `(10)))
(!softmax a)
;#Const((0.910... 0.886... ~ 0.802... 0.616...) :mgl t :shape (10))
2
Softmax is applied to dim=0 
(setq a (!randn `(10 10)))
;#Const(((-0.29... -1.99... ~ -0.36... 1.725...)        
;                 ...
;        (0.695... -0.94... ~ 1.179... 0.655...)) :mgl t :shape (10 10))

(!softmax a)
;#Const(((0.064... 0.011... ~ 0.060... 0.489...)        
;                 ...
;        (0.129... 0.024... ~ 0.209... 0.124...)) :mgl t :shape (10 10))
3
Softmax is applied to dim=0 
(setq a (!randn `(10 10 10)))
;#Const((((2.585... 0.517... ~ 0.428... 0.059...)         
;                   ...
;         (-2.11... 0.308... ~ -0.91... 0.649...))        
;                 ...
;        ((-0.75... 1.030... ~ 0.656... -0.00...)         
;                   ...
;         (-0.37... -0.52... ~ 1.589... -0.10...))) :mgl t :shape (10 10 10))

(!softmax a)
;#Const((((0.374... 0.047... ~ 0.043... 0.029...)         
;                   ...
;         (0.010... 0.115... ~ 0.033... 0.162...))        
;                 ...
;        ((0.029... 0.172... ~ 0.118... 0.061...)         
;                   ...
;         (0.048... 0.041... ~ 0.345... 0.063...))) :mgl t :shape (10 10 10))
4
Todo: currently, it returns error.

with-verbose

with-verbose(&body body)
In the codes below, the computation nodes will be displayed when (backward out)

with-dtype

with-dtype(dtype &body body)
Switches the dtype. dtype = (:float :double). In default, :float.

dtypecase

dtypecase(&rest cases)
todo :docstring

define-with-typevar

define-with-typevar(function-name type-specifier (&rest args) &body body &aux (fnames (map (quote list) (function (lambda (p) (symb function-name p))) *dtype-prefixes*)) (params (get-params args)))
Todo: Document

with-backend

with-backend(backend &body body)

Switches a backend.

See also: define-node-extension

define-node-extension

define-node-extension(name &key optimize backend (disassemble-forward nil) forward-declaim forward (disassemble-backward nil) backward-declaim backward)

Adds a new backend to the defined node.

The type of backend is managed by keywords. The backend defined in defnode is always :mgl.

Defined backends can be switched by the macro (with-backend backend).

As long as *restart-non-exist-backend* is t, when a computation node reaches a backend that is not defined, :mgl is called, otherwise the condition backend-doesnt-exists will occurs.

Example:

(define-node-extension cl-waffe::AddTensor
  :backend :test-backend
  :forward ((x y)
        (const (+ 1 1)))
  :backward ((dy)
         (list dy dy)))

(with-backend :mgl
   (print (!add 10 10))) ;=> Const(20)

(with-backend :test-backend
   (print (!add 10 10))) ;=> Const(2)

(with-backend :hogehoge
   (print (!add 10 10))) ; => Const(20)

(let ((*restart-non-exist-backend* nil))
    (with-backend :hogehoge
        (print (!add 10 10)))) ;=> Evaluation aborted on #<CL-WAFFE::BACKEND-DOESNT-EXISTS {100FA18C43}>.

*restart-non-exist-backend*

*restart-non-exist-backend*
When t, in the case when the specified backend doesn't exist, cl-waffe calls a standard implementation backend

!allow-destruct

!allow-destruct(tensor)

Tensors which path through this macro are allowed to be destructed by cl-waffe's kernel.

In default, cl-waffe's operators won't make side effects. 

(setq a (!randn `(3 3)))

;#Const(((0.811... -0.43... -0.91...)        
;                 ...
;        (0.959... -0.62... 1.150...)) :mgl t :shape (3 3))

(!exp a)
;#Const(((2.252... 0.645... 0.400...)        
;                 ...
;        (2.610... 0.534... 3.159...)) :mgl t :shape (3 3))

(print a)
;#Const(((0.811... -0.43... -0.91...)        
;                 ...
;        (0.959... -0.62... 1.150...)) :mgl t :shape (3 3))

However, This macro let kernel know that the given tensor is allowed to destruct(i.e.: the result is overwritten)

(setq a (!randn `(3 3)))

;#Const(((0.811... -0.43... -0.91...)        
;                 ...
;        (0.959... -0.62... 1.150...)) :mgl t :shape (3 3))

(!allow-destruct a)
; T

(!exp a)
;#Const(((2.252... 0.645... 0.400...)        
;                 ...
;        (2.610... 0.534... 3.159...)) :mgl t :shape (3 3))

(print a) ; You can see the result is overwritten.
;#Const(((2.252... 0.645... 0.400...)        
;                 ...
;        (2.610... 0.534... 3.159...)) :mgl t :shape (3 3))

Avoiding copy, destructive operations are superior in terms of memory usage.

(setq a (!randn `(100 100)))

(time (!exp a))
;Evaluation took:
;  0.000 seconds of real time
;  0.000275 seconds of total run time (0.000219 user, 0.000056 system)
;  100.00% CPU
;  498,150 processor cycles
;  31,264 bytes consed

(!allow-destruct a)

(time (!exp a))
; Evaluation took:
;  0.000 seconds of real time
;  0.000178 seconds of total run time (0.000160 user, 0.000018 system)
;  100.00% CPU
;  273,646 processor cycles
;  0 bytes consed

See also: !disallow-destruct which does the opposite.

!disallow-destruct

!disallow-destruct(tensor)
Tensors that path through this macro are not destructed.
(setq a (!randn `(3 3)))
;#Const(((1.084... -1.10... 1.406...)        
;                 ...
;        (1.044... 0.059... -0.53...)) :mgl t :shape (3 3))

(!allow-destruct a)
; T
(!disallow-destruct a)
; NIL

(!exp a)
;#Const(((2.957... 0.329... 4.080...)        
;                 ...
;        (2.840... 1.060... 0.584...)) :mgl t :shape (3 3))

(print a) ; a is kept remained.
;#Const(((1.084... -1.10... 1.406...)        
;                 ...
;        (1.044... 0.059... -0.53...)) :mgl t :shape (3 3))

defnode

defnode(name initializer-arguments &key parameters (disassemble-forward nil) forward-declaim forward (disassemble-backward nil) backward-declaim backward (document An node, defined by cl-waffe.))

Defines computation nodes in a format that cl-waffe can handle.

Note: the data structures that can be used in arguments, and returned values, must be following:

  1. WaffeTensor
  2. 1D list which each element is WaffeTensor

Be aware that you can't use (values x y ...).

name
The node's name. constructor and structure are being defined named after this argument.
initializer-argument
arguments the constructor have.
parameter
The parameters this node has being initializer with initializer-argument.
disassemble-forward
when t, when this node is compiled, display the disassemble of forward slot.
forward-declaim
Describe the declaim for the forward function. Note that the first argument is a structure. and :forward keyword in this declaim will be replaced by the forward function's name.
forward
the definition of forward
disassemble-backward
when t, when this node is compiled, display the disassemble of backward slot.
backward-declaim
Describe the declaim for the backward function. Note that the first argument is a structure. and :backward keyword in this declaim will be replaced by the backward function's name.
backward
the definition of backward

defmodel

defmodel(name initializer-arguments &key (parameters nil) (disassemble-forward nil) forward-declaim forward (document An model, defined by cl-waffe))

This macro defines a cl-waffe model as name.

At the same time, a constructor name is defined and you can initialize your model like:

(cl-waffe.nn:LinearLayer 100 20) ; => [Model: Linearlayer]

name
Your model and constructor name
args
The arguments of a constructor
parameters

The parameters your model has.

Every time you initialize the model, the parameters are initialized.

Note that defmodel behaves like class.

The arguments are the same as defstruct

Format Example: ((param-name param-initial-value &key (type your-type)))

forward

Define here the forward propagation of your model.

When backward, Automatic differentiation applies.

defoptimizer

defoptimizer(name initializer-arguments &key parameters (disassemble-update nil) update-declaim update (document An optimizer, defined by cl-waffe.))

Defines optimizer in the format that cl-waffe can handle.

Name
The optimizer's structure and constructor will be defined after name
Args
Initializer of the optimizer. The first value of initializer is the hash-table that collected model's parameter where the key is fixnum from 0 to n. You have to store it.
parameters
An parameters that it has.
update
when training and (update) is called, this slot is called and you optimizer your parameters.
optimize
when t, the :update slot is defined with (optimize (speed 3)(space 0)(debug 0)) Default: nil
document
docstring for optimizers. You can use string or (with-usage) macro

Example:

;defoptimizer's args must start with params (symbol-name doesn't matter) which receives hash-table whose key is 1..n

(defoptimizer SGD (params &key (lr 1e-3))
  :optimize t
  :parameters ((params params :type hash-table)
               (lr lr :type single-float))
  :update (()
       (dotimes (i (hash-table-count (self params)))
         ; W(n+1) = W(n) - n * grad
         (!modify (gethash i (self params))) :+=
               (!mul (self lr)(grad (gethash i (self params)))))))

;(call (SGD (find-variables model))) will works as update.
;(call-backward (SGD)) will works as zero-grads.

call

call(model &rest inputs &aux (features (model-inlineable-p model)))
calls the given model's forward slot with inputs.

call-backward

call-backward(model &rest inputs)
calls the given model's backward, with inputs.

self

self(name)
Todo: Docstring

save-for-backward

save-for-backward(slot tensor)
TODO :DOCSTRING

get-forward-caller

get-forward-caller(model)
Returns the given node (model/node/optimizer)'s forward slot, which is callable with funcall/apply.

get-backward-caller

get-backward-caller(model)
Returns the given node (model/node/optimizer)'s backward slot, which is callable with funcall/apply.

with-calling-layers

with-calling-layers(input &rest layers)

This macro allows to sequentially call layers.

the argument input must be a tensor.

Refering each layers from (self) macro, destructively modifying x with the returned value.

Note: This macro supposes models to be returned a single tensor, not a list.

(defmodel MLP (activation)
   :parameters ((layer1   (denselayer (* 28 28) 512 T activation))
   	        (layer2   (denselayer 512 256 T activation))
	        (layer3   (linearlayer 256 10 T)))
   :forward ((x)
	     (with-calling-layers x
	       (layer1 x)
 	       (layer2 x)
               (layer3 x))))

For the different arguments.

(with-calling-layers x
     (layer1 x 1 1)
     (layer2 1 x 2)
     (layer3 x y))

Output: An last value of layers.

deftrainer

deftrainer(name args &key model optimizer optimizer-args step-model predict (document An trainer structure defined by cl-waffe.))

Defining trainer, which is made in order to call train function.

The slots you defined can be invoked by using (step-model model &rest args), (predict model &rest args). See below.

model
An model defined by (defmodel) which you want to train.
optimizer
An optimizer defined by (defoptimizer)
optimizer-args
An arguments for optimizer
step-model
For each batch step, :step-model is called in (train) function. Describe here forward step, backward, zero-grad, update for training.
predict
an code for predicting

These macro below are defined by macrolet and you can use them in :step-model, :predict

(self name)
access trainer's parameters.
(model)
access trainer's model, defined by :model keyword.
(zero-grad)
Find model's all parameters and constants, and initialize their grads. (i.e. call optimizer's backward)
(update)
Find model's all parameters, and call optimizer and change parameter's data. (i.e. call optimizer's forward)

This trainer macro is defined in order to integrate following works:

  1. calling models
  2. calling criterions
  3. calling backward
  4. calling optimizer
  5. calling zero-grad
  6. defining predict

Example:

(deftrainer MLPTrainer (activation lr)
  :model          (MLP activation)
  :optimizer      cl-waffe.optimizers:Adam ; Note: :optimizer requires a single variable.
  :optimizer-args (:lr lr) ; these arguments directly expanded to optimizer's args.
  :step-model ((x y)
	       (zero-grad) ; call zero-grad
	       (let ((out (cl-waffe.nn:softmax-cross-entropy (call (model) x) y))) ; get criterion
		 (backward out) ; backward
		 (update) ; call optimizer
		 out)) ; return loss
 :predict ((x)(call (model) x))) ;for predict

(setq trainer (MLPTrainer :relu 1e-4)) ; init your trainer

; Train:   (step-model trainer model-input-x model-input-y)
; Predict: (predict trainer model-input-x)

step-model

step-model(trainer &rest args)

An function for calling trainer object defined by deftrainer By using this function, trainer's step-model will be invoked.

Input: Trainer, Args

predict

predict(trainer &rest args)
An function for calling trainer's predict slot

model

modelnil
Todo: Docstring

update

update(&rest args)
Todo: Docstring

zero-grad

zero-gradnil
Todo docstring

defdataset

defdataset(name args &key parameters next length (document An dataset structure defined by cl-waffe.))

Defining dataset. (This is kinda pytorch's dataloader)

The slots you defined can be invoked by using (get-dataset dataset index)(get-length dataset).

parameters
parameters datasets have.
next
when function (get-dataset dataset index) is called, this slot invokes. Return waffetensor for the next batch in response to your task.
length
In this form, the function must return the total length of your datasets where the value is fixnum. (Not a batch, and not a current index.)
(defdataset Mnistdata (train valid batch-size)
  :parameters ((train train)(valid valid)(batch-size batch-size))
  :next    ((index)
	    (list (!set-batch (self train) index (self batch-size))
		  (!set-batch (self valid) index (self batch-size))))
  :length (()(car (!shape (self train)))))

cl-waffe excepts index to be 1, 2, 3, ... (dataset-maxlen)

So, please manage batch-sizes in args and :next slots.

get-dataset

get-dataset(dataset index)
Get datum of the index from dataset. Input: dataset ... dataset defined by defdataset. index ... fixnum

get-dataset-length

get-dataset-length(dataset)
Get total size of your dataset.

model-list

No node with name model-list.

mlist

mlist(&rest models)
define mlist

mth

mth(index mlist)
Accessor for model-list

grad

grad(tensor)

Accessing tensor's grad.

When tensor's grad is nil, an error occurs

Input
WaffeTensor
Output
An tensor's grad which is the type of mgl-mat:mat or waffetensorcontettype

Note: grad is not setfable